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Data  @ Google  is  BIG 

• Google’s  main  applications  are  their  search  and  mail 

• The  entire  Internet  and  everyone’s  mail  is  a lot  of  data 

• Traditional  data  storage  approaches  such  as  a relational 
database  just  don’t  scale  to  the  size  at  which  Google 
applications  operate 

• Sharded,  sorted,  array  with  hierarchical  keys 

http://labs.google.com/papers/bigtable.html 
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Under  the  Covers  of  the  Google  App  Engine  Datastore 

Ryan  Barrett  (Google) 


Ever  wonder  why  you  cant  do  joins  in  the  Google  App  Engine  datastore?  Why  your  app  is  seeing 
deadlines  so  often?  Why  it's  so  hard  to  tell  whether  a query  will  need  an  index?  Why  we  offer  both 
parent/child  relationships  and  reference  properties?  Or  why  list  properties  dont  seem  to  make  any 
sense  at  all?  This  talk  will  explain  how  the  datastore  itself  works,  why  these  seeming  peculiarities 
(and  many  others!)  exist,  and  what  you  can  do  about  them. 
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http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore 


Model-View-Controller 

Design  Pattern 


Tasks  Inside  the  Server 

• Process  any  form  input  - possibly  storing  it  in  a database  or 
making  some  other  change  to  the  database  such  as  a delete 

• Decide  which  screen  to  send  back  to  the  user 

• Retrieve  any  needed  data 

• Produce  the  HTML  response  and  send  it  back  to  the  browser 
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Model  View  Controller 

• We  name  the  three  basic  functions  of  an  application  as 
follows 

• Controller  - The  Python  code  that  does  the  thinking 
and  decision  making 

• View  - The  HTML,  CSS,  etc.  which  makes  up  the  look 
and  feel  of  the  application 

• Model  - The  persistent  data  that  we  keep  in  the  data 
store 

http://en.wikipedia.org/wiki/Model-view-controller 


Terminology 

• We  call  the  Data  bit  - the  “Model”  or  Data  Model 

• We  call  the  “making  the  next  HTML”  bit  the  “View”  or 
“Presentation  Layer” 

• We  call  the  handling  of  input  and  the  general  orchestration  of  it 
all  the  “Controller” 


Model-View-Controller 


“In  /VI  VC,  the  model  represents  the 
information  (the  data)  of  the  application 
and  the  business  rules  used  to 
manipulate  the  data;  the  view 
corresponds  to  elements  of  the  user 
interface  such  as  text,  checkbox  items, 
and  so  forth;  and  the  controller  manages 
details  involving  the  communication  to 
the  model  of  user  actions.’’ 


http://en.wikipedia.org/wiki/Model-View-Controller 
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http://www.kapralova.org/MORAL.htm 


The  controller  is  the  conductor  of  all  of  the  other  aspects  of  MVC. 


Our  Architecture:  MVC 

• Model  - Holds  the  permanent  data  which  stays  long  after  the 
user  has  closed  their  web  browsers 

• View  - Produces  the  HTML  Response 

• Controller  - Receives  each  request  and  handles  input  and 
orchestrates  the  other  elements 


Adding  Models  to  our 
Application 

ae- 1 0-datastore 


http://code.google.com/appengine/docs/datastore/ 


django 


Django  Models 


• Thankfully  we  use  a very  simple  interface  to  define 
objects  (a.k.a.  Models)  and  store  them  in  BigTable 

• Google's  BigTable  is  where  the  models  are  stored 

• We  don’t  need  to  know  the  details 

• The  pattern  of  these  models  is  taken  from  the  Django 
project 

http://docs.djangoproject.com/en/dev/ref/models/instances/?from=olddocs 


Property  Types 

• StringProperty  - Any  string 

• IntegerProperty  - An  Integer  Number 

• DateTimeProperty  - A date  + time 

• BlobProperty  - File  data 

• ReferenceProperty  - A reference  to  another  model 
instance 

http://code.google.com/appengine/docs/datastore/ 


A Simple  Model 

from  google.appengine.ext  import  db 

#A  Model  for  a User 

Each  model  is  a 

class  User(db.Model): 

Python  class  which 

acct  = db.StringProperty() 

extends  the 

pw  = db.StringProperty() 
name  = db.StringProperty() 

db.Model  class. 

newuser  = User(name=”Chuck”, 
newuser.put() 

acct=”csev”,  pw=”pw”) 

Property  class 

Value  type 

Sort  order 

StringProperty 

str 

Unicode 

Unicode  (str  is  treated  as  ASCII) 

BooleanProDertv 

bool 

False  < True 

IntegerProperty 

int 

Numeric 

lono 

FloatProDertv 

float 

Numeric 

DateTimeProperty 

DateProDertv 

TimeProoertv 

datetime.datetime 

Chronological 

ListProperty 

StrinaListProDertv 

list  of  a supported  type 

If  ascending,  by  least  element;  if  descending,  by  greatest  element 

ReferenceProperty 

SelfReferenceProoertv 

db.Kev 

By  path  elements  (kind,  ID  or  name,  kind,  ID  or  name...) 

UserProDertv 

users.  User 

By  email  address  (Unicode) 

BlobProDertv 

db.Blob 

(not  orderable) 

TextProoertv 

db.Text 

(not  orderable) 

Category  Property 

db.  Category 

Unicode 

Keep  it  simple 

for  a while 

from  google.appengine.ext  import  db 

#A  Model  for  a User 

Each  model  is  a 

class  User(db.Model): 

Python  class  which 

acct  = db.StringProperty() 

extends  the 

pw  = db.StringProperty() 
name  = db.StringProperty() 

db.Model  class. 

newuser  = User(name=”Chuck”,  acct=”csev”,  pw=”pw”); 
newuser.put(); 

class  ApplyHandler(webapp.RequestHandler): 


App  Engine  - HTML 

0 http://localhost:8080/apply  © • Google 


def  post(self): 

self.session  = Session() 
xname  = self.request.get('name') 
xacct  = self.request.get('account') 
xpw  = self.request.get('password') 


Get  Session 

Form 

Data 


# Check  for  a user  already  existing 

que  = db.Query(User).filter("acct  =",  xacct) 

results  = que.fetch(limit=  I) 


App  Engine  Sites  Topics  Login 

New  Account  Request 

Please  enter  your  information  below: 

Name:  chuck 
Account: 


Password:  .. 


if  len(results)  > 0 : Check  for 

doRender(self, "apply.htm", {'error'  : 'Account  Already  Exists'} ) existing  USer 
return  ° 


newuser  = User(name=xname,  acct=xacct,  pw=xpw);  Insert  User 

newuser.put(); 

self.session['username']  = xacct  . . , - 

doRender(self, "index.htm", { })  Update  beSSIOn 


App  Engine  - HTML 


0http://localhost:8O8O/login 

© -fCL*  Google 

» 

App  Engine  Sjtes  Topics  Login 


Please  Log  In 

Please  enter  your  id  and  password  to  log  in  to  this  site. 


Account: 

Password: 

( Submit ) ( Cancel  ) ( New  Account 


App  Engine  - HTML 

0 http://localhost:8080/apply  © »(<V  Google  '(j* 


App  Engine  Sites  Topics  Login 


Inserting  a User 
and  listing  Users 


New  Account  Request 

Please  enter  your  information  below: 

Name:  chuck 
Account:  ^ 


Password:  •• 


App  Engine  - HTML 

C r*http://localhost:8080/memb<©  Google  ^ jj 


App  Engine  Sites  Topics  Members  Logout (csev) 


Members 


\ 


Name  Account  Password 


|Chuck|csev  |pw 


http://localhost:8080/_ah/admin/ 


Using  the  Developer 
console  we  can  see  the 
results  of  the  put() 
operation  as  a new  User 
object  is  now  in  the  data 
store. 


ae-10-datastore  Development  Console  - Datastore  Viewer 
■*  6 + 1 <http://localhost:8080/_ah/admin/datastore?kind=U©  * CV 


Google  App  Engine 


it  Conso^^ 

'iev^r 


ae-10-datastore  Development 
Datastore  Vi 


!W  Entity) 


Memcaehe  Viewer 


newuser  = User(name=xname,  acct=xacct,  pw=xpw); 
newuser.putQ; 


class  MembersHandler(webapp.RequestHandler): 

def  get(self): 

que  = db.Query(User) 
user_list  = que.fetch(limit=IOO) 
doRender(self,  'memberscreen.htm', 
{'user_list':  user_list}) 


We  simply  construct  a query  for  the  User  objects,  and  fetch 
the  first  100  User  Objects.  Then  we  pass  this  list  into  the 
memberscreen.htm  template  as  a context  variable  named 
‘user  list’. 


Google  App  Engine 
References 
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{%  extends  "_base.htm"  %} 
{%  block  bodycontent  %} 
<h  1 >Members</h  1 > 

<p> 

<table> 

templates/members.htm 

<tr><th>Name</th><th>Account</th><th>Password</th></tr> 
{%  for  user  in  user_list  %} 

<tr> 

<td>{{  user.name  }}</td> 

<td>{{  user.acct  }}</td> 

In  the  template,  we  use  the  for 

<td>{{  user.pw  }}</td> 

directive  to  loop  through  each 

</tr> 

user  in  the  userjist  variable  in 
the  context.  For  each  user  we 

{%  endfor  %} 

construct  a table  row  with  their 

</table> 

{%  endblock  %} 

name,  account,  and  pw. 

Relationships 

• We  need  to  create  a new  model  for  Chat  messages  and 
then  relate  Chat  messages  by  marking  them  as 
belonging  to  a particular  user 


•<  C <5  + «http://localhost:8080/chat  <?  - Q.- 

App  Engine  Sites  Topics  Chat  Members  Logout  (sally) 

Appengine  Chat 

1 ( Submit ) 

Yes,  it  was  surprisingly  easy  - make  sure  to  look  at  the  key()  method  (sally)  Sat  22  Nov  2008 
Have  you  used  a Reference  yet?  (csev)  Sat  22  Nov  2008 
Hi  there  (sally)  Sat  22  Nov  2008 


Chat  msg 
Chat  msg 
Chat  msg 


User  csev 


User  sally 


class  User(db.Model): 

acct  = db.StringProperty() 
pw  = db.StringProperty() 
name  = db.StringProperty() 


User 


acct 


pw 


name 


Three  Kinds  of  Keys 

• Logical  Key  - What  we  use  to  look  something  up  from 
the  outside  world  - usually  unique  for  a model 

• Primary  Key  - Some  “random”  number  which  tells  the 
database  where  it  put  the  data  - also  unique  - and 
opaque 

• Reference  - When  we  have  a field  that  points  to  the 
primary  key  of  another  model  (a.k.a.  Foreign  Key) 


class  User(db.Model): 

acct  = db.StringProperty() 
pw  = db.StringProperty() 
name  = db.StringProperty() 


newuser  = User(name=name,  acct=acct,  pw=pw) 
newuser.put() 

self.sessionfusername']  = acct 
self.sessionfuserkey']  = newuser.key() 


User 


class  User(db.Model): 

acct  = db.StringProperty() 
pw  = db.StringProperty() 
name  = db.StringProperty() 


newuser  = User(name=name,  acct=acct,  pw=pw) 
key  = newuser.put(); 
self.session['username']  = acct 
self.session['userkey']  = key 


When  we  log  in... 

que  = db.Query(User).f)lter("acct  =",acct).filter("pw  = ",pw) 
results  = que.fetch(limit=l) 
if  len(results)  > 0 : 
user  = res  u Its  [0] 
self.sessionfusername']  = acct 
self.sessionfuserkey']  = user.key() 
doRender(self, "index.htm", { } ) 
else: 

doRender(self, "loginscreen.htm", 

{'error'  : 'Incorrect  login  data'}  ) 


Fast  Lookup  By  Primary  Key 


• Lookup  by  primary  key  is  faster  than  by  logical  key  - 
because  the  primary  key  is  about  “where”  the  object  is 
placed  in  the  data  store  and  there  is  *only  one* 

• So  we  put  it  in  session  for  later  use... 

newuser  = User(name=name,  acct=acct,  pw=pw); 
key  = newuser.put(); 
self.session['username']  = acct 
self.sessionfuserkey']  = key 


When  we  log  Out... 

class  LogoutHandler(webapp.RequestHandler): 

def  get(self): 

self.session  = Session() 
self.session.delete_item('username') 
self.session.delete_item('userkey') 
doRender(self,  'index.htm') 

When  we  log  out  - we  make  sure  to  remove  the  key  from  the  session 
as  well  as  the  account  name. 


Making  References 


References 


• When  we  make  a new  object  that  needs  to  be 
associated  with  or  related  to  another  object  - we  call 
this  a “Reference” 

• Relational  Databases  call  these  “Foreign  Keys” 


App  Engine  - HTML 

◄ | ► 6 + 0 http://localhost:8080/chat  Q - ''Q.*  Google 

App  Engine  Sites  Topics  Chat  Members  Logout (sallvt 

Appengine  Chat 

( Submit ) 

Yes,  it  was  surprisingly  easy  - make  sure  to  look  at  the  key()  method  (sally)  Sat  22  Nov  2008 
Have  you  used  a Reference  yet?  (csev)  Sat  22  Nov  2008 
Hi  there  (sally)  Sat  22  Nov  2008 


Database 

Normalization 


We  could  just  store  the  account  strings  in  each  chat  message.  This  is 
bad  practice  generally  - particularly  if  we  might  want  to  know  more 
detail  about  the  User  later.  We  don’t  like  to  make  multiple  copies  of 
anything  except  primary  keys. 

http://en.wikipedia.org/wiki/Database_normalization 


class  ChatMessage(db.Model): 
user  = db.ReferenceProperty() 
text  = db.StringProperty() 

created  = db.DateTimeProperty(auto_now=True) 


So  we  make  a reference  property  in  our  Chat  message  model.  The  property 
does  *not*  need  to  be  named  “user”  - but  it  is  a convienent  pattern.  Also  note 
the  created  field  that  we  let  the  data  store  auto-populate. 


class  ChatMessage(db.Model):  Populating 

user  = db.ReferenceProperty()  References 

text  = db.StringProperty() 

created  = db.DateTimeProperty(auto_now=True) 

def  post(self): 

self.session  = Session() 

msg  = self.request.get('message') 

newchat  = ChatMessage(user  = self.session['userkey'],text=msg) 
newchat.put(); 

When  we  create  a ChatMessage,  we  get  the  message  text  from  the  chatscreen.htm 
form,  and  then  user  reference  is  the  key  of  the  current  logged  in  user  taken  from  the 
Session.  Note:  Some  error  checking  removed  from  this  example. 


Relating 

Models 

User 

ChatMessage 

key() 

V 

key() 

class  User(db.Model): 

acct 

user 

acct  = db.StringProperty() 
pw  = db.StringProperty() 
name  = db.StringProperty() 

pw 

text 

name 

created 

class  ChatMessage(db.Model): 
user  = db.ReferenceProperty() 
text  = db.StringProperty() 

created  = db.DateTimeProperty(auto_now=True) 

App  Engine  - HTML 

| ► 6 M http://localhost:8080/chat  © - ^Q.-  Coogle 

App  Engine  Sites  Topics  Chat  Members  Logout  (sally) 

Appengine  Chat 


Yes,  it  was  surprisingly  easy  - make  sure  to  look  at  the  key()  method  (sally)  Sat  22  Nov  2008 
Have  you  used  a Reference  yet?  (csev)  Sat  22  Nov  2008 
Hi  there  (sally)  Sat  22  Nov  2008 

) 


We  need  to  display  the  list  of  the  most  recent 
ChatMessage  objects  on  the  page. 


def  post(self): 

self.session  = Session() 

msg  = self.request.get('message') 

newchat  = ChatMessage(user  = self.session['userkey'],  text=msg) 
newchat.put(); 

que  = db.Query(ChatMessage).order("-createdM); 
chatjist  = que.fetch(limit=  10) 
doRender(self, "chatscreen.htm", 

{ 'chatjist':  chatjist }) 

We  retrieve  the  list  of  chat  messages,  and 
pass  them  into  the  template  as  context 
variable  named  “chatjist”  and  then 
render  “chatscreen.htm”. 


ChatMessage 


{%  extends  "J3ase.htm"  %} 

{%  block  bodycontent  %} 

<h  I >Appengine  Chat</h  I > 

<P> 

<form  method="post"  action=7chat"> 

<input  type="text"  name="message"  size="60"/> 
<input  type="submit"  name="Chat7> 

</form> 

</p> 

{%  ifnotequal  error  None  %} 

<P> 

{{  error }} 

</p> 

{%  endifnotequal  %} 

{%  for  chat  in  chatjist  %} 

<p>{{  chattext }}  ({{chat.user.acct}}) 
{{chat.created|date:"D  d MY"}}</p> 

{%  endfor  %} 

{%  endblock  %} 


chatscreen.htm 


In  the  chatscreen.htm 
template,  we  loop  through 
the  context  variable  and 
process  each  chat  message. 

For  a reference  value  we 
access  the  .user  attribute  and 
then  the  .acct  attribute 
within  the  .user  related  to 
this  chat  message. 


chatscreen.htm 


{%  extends  "_base.htm"  %} 

{%  block  bodycontent  %} 

<h  I >Appengine  Chat</h  I > 

<P> 

<form  method="post"  action=7chat"> 

<input  type="text"  name="message"  size="607> 
<input  type="submit"  name="Chat7> 

</form> 

</p> 

{%  ifnotequal  error  None  %} 

<P> 

{{  error  }} 

</p> 

{%  endifnotequal  %} 

{%  for  chat  in  chatjist  %} 

<p>{{  chattext }}  ({{chat.user.acct}}) 
{{chat.created|date:"D  d MY"}}</p> 

{%  endfor  %} 

{%  endblock  %} 


In  the  chatscreen.htm 
template,  we  loop  through 
the  context  variable  and 
process  each  chat  message. 


Walking  a reference 

• The  chatjist  contains  a list  of  chat  objects 

• The  iteration  variable  chat  is  each  chat  object  in  the  list 

• chat.user  is  the  associated  user  object  (follow  the  reference) 

• chat.user.acct  is  the  user’s  account 

{%  for  chat  in  chatjist  %} 

<p>{{  chat.text }}  ({{chat.user.acct}}) 
{{chat.created|date:"D  d MY"}}</p> 

{%  endfor  %} 


ae-ll-chat  Development  Console  - Data 


■*  + Jg^ttp7/localhost:8O8O/_^/^mln/d^smre/rt|^^^^0^^C^I^^^J 

Google  App  Engine 

ae-11-chat  Development  Console 
Edit  Entity 

natactnra  Uinu/Ar  # 


ae-ll-chat  Development  Console  - Datas 


chat 


Interactive  Console 
Memcache  Viewer 
Task  Queues 
Cron  Jobs 
XMPP 

Entity 

Kind 

Entity 

Key 

ID 

agphZS0xMS1jaGF0chELEgtDaGF0TWVzc2FnZRglDA 

37  chattext 

Inbound  Mail 

text 

Have  you  used  a Reference  yet? 

user 

agphZS0xMS1jaGF0cgoLEgRVc2VyGAgM 

created 

2008-11-22  13:49:55 

(Save Changes)  (SST)  Chat.USGF 

a 

©2009  Google 

■*  <5  + f^nttp://localhost:8080/_ah/admin/datastore/edu?key  C 1 Q,*  Google 

Google  App  Engine 

ae-1 1-chat  Development  Console 
Edit  Entity 


chat.user 


Entity  agphZS0xMS1jaGF0cgoLEgRVc2VyGAgM 

r „ chat.user.acct 


i 


account  Csev 


(string) 

(string) 


(Save  Changes)  ( Delete  ) 


{%  for  chat  in  chatjist  %} 

<p>{{  chat.text }}  ({{chat.user.acct}}) 
{{chat.created|date:"D  d MY"}}</p> 
{%  endfor  %} 


Summary 

• All  objects  stored  in  the  data  store  are  given  a primary 
key  which  we  get  from  either  the  put()  call  or  the  key() 
call 

• We  place  these  keys  in  ReferenceProperty  values  to 
connect  one  model  to  another 

• When  an  attribute  is  a reference  property,  we  use 
syntax  like  chat.user.acct  - to  look  up  fields  in  the 
referenced  object 


chatscreen.htm 


{%  extends  "_base.htm"  %} 

{%  block  bodycontent  %} 

<h  I >Appengine  Chat</h  I > 

<P> 

<form  method="post"  action=7chat"> 

<input  type="text"  name="message"  size="607> 
<input  type="submit"  name="Chat7> 

</form> 

</p> 

{%  ifnotequal  error  None  %} 

<P> 

{{  error  }} 

</p> 

{%  endifnotequal  %} 

{%  for  chat  in  chatjist  %} 

<p>{{  chat.text }}  ({{chat.user.acct}}) 
{{chat.created|date:"D  d MY"}}</p> 

{%  endfor  %} 

{%  endblock  %} 


To  make  the  date  format  a 
little  nicer  we  use  a |date: 
formatter  which  shows  the 
day  of  week,  day  of  month, 
month,  and  year. 


